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ABSTRACT 

This speech discusses the planning for and the 
implementation of the 0#E#0# . performance contracting experiments* The 
author argues that it was not performance contractingr but the 0.E*0. . 
experiment which failed* .The author blames this failure not so much 
on the poor evaluation design, but rather on the ntimber of interface 
problems which arose because of the conflict between experiment 
design and project implementation* (Author/JF) 
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On February 1st a year ago, O.E.O. announced the preliminary results 
of its unique large-scale experiment with performance contracting — that 
"performance contracting had failed to raise student achievement in math 
and reading." A year later to that date, based upon a prematurely released 
U.S.O.E. study, headlines again reported "that the offer of incentives to 
teachers did not increase student performance." Ironically, and paradox- 
ically, on that same day. Dr. John Porter, Chief State School Officer of 
Michigan, and I appeared before the House of Labor and Education Committee, 
as the first private witnesses testifying on the Elementary and Secondary 
Education Act extension. At that tiine Dr. Porter released the results of 
his state-wide $23 million Performance Contract- Accountability Model, in- 
volving 66 districts and 112,000 students, indicating that ^3% of the stu- 
■» dents achieved their objectives in math; the state-vride "caro ed" average 
grade equivalent gain per student in math and reading was approximately 
1e^ grade gain. I also reported on the results of the Dade County Incen- 
tives Project where students averaged between 50 and 300^ above expected 
gain and a performance contracting project conducted in Grand Rapids, 
Michigan, where 150 educable mentally retarded (EMR) students with an 
average I.Q. of 59 » achieved over a month's gain for a month in the pro- 
gram in math and reading. And two days prior to that testimony, a lower 
Federal court in Texarkana, Arkansas, after several days of hearings, 
found that Dorsett Education Systems didn't teach test items and awarded 
the firm $30,000 which it sought as the result of the allegations made 
by Education Testing Service, and others in the infamous Texarkana Per- 
formance Contract Project. 

Is there any reason why educators and laymen are confused about per- 
^ formance contracting? Having been intimately involved in all of the above 
^ projects, I can assure you that the fiction written thus far about these 
Q projects is probably more believable than the reality of what has happened.. 

If^ The questions which I have been asked to address are, "Did the Federal 

O Government fail performance contracting, and if so, how and why?." In the 
O final analysis you and the general public must decide the ansv/er to the 
O ^ first question when all the facts are presented. Hopefully, while offering 
ERIC M some constructive criticism regarding the questions of "how", I will focus 
" on two questions: 
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1) Was the concept of performance contracting and turnkey operations 
misused; and 

2) What Was actually being tested ~ the ability of the Federal agen- 
cies to conduct experiments, or the technique? 

The question of intent, particularly O.E.O.«s can only be hyoothesized 
at this time, 

Background of the O.E.O. and Belated Experiments 

^ ^ kP® 0»E.O. project officially began in the late spring, 1970, with the 
inteht of testing whether different instructional systems used by various 
contracting firms could increase significantly math and reading achievement 
of deficient children. Approximately 30,000 students (X,C, and comparison 
groups ^ m 20 districts, 6 firms and two teacher groups, an evaluation con- 
tractor, a relatively large O.S.O. staff, and a management support group 
were involved in this $8-10 million project. On January 31, 1972, O.E.O. 
Director Philip Sanchez announced that "we have not found the panacea." 
Preliminary comparisons of experimental vs. control (E vs. C) groups by 
grade level "resulted in no significant differences." And since O.E.O. 
felt it "could perform a valuable public service to those school boards 
(considering performance contracting) by subjecting the claims of perform- 
ance contractors to scientific scrutiny before these arrangements became 
fixed in concrete," O.E.O. appeared satisfied with the job it had done. 
Subsequently, Battelle Memorial Institute, the evaluation contractor, sub- 
mitted at least two additional reports and O.E.O., some six to eight months 
later, issued its final report without any significant changes. All O.E.O. 
claims were challenged! 

The U.S.O.E. Incentives Project, conducted in four districts during 
school year 1 971 -72, intended to determine what, if anything teachers and 
parents would do different when offered financial incentives, individually, 
based upon student performance. Evaluation reports indicate that few sig- 
nificant changes in teacher and parental educational activity occurred 
and student performance did not change significantly. In the Texarkana 
experiment, while preliminary results in March, 1970 indicated student 
performance significantly greater than expected performance, the allegations " 

of "teaching test items" precluded public disclosure of the results 

even those allegedly contaminated. 

Misuse of the Performance C ontracting-Turnkev Concept? 

Three years before this annual convention, as I discussed the perform- ^ 
ancG contract turnkey approach, I emphasized that it should be viewed as j 
a means to an end, namely school system reform, rather than an end itself; ^ 
that as a tool it v/as limited technically and administratively and was 1 
subject to the intentions of man; and that it offered a low-risk. low-cost i 
means for experimentation, a politically feasible and acceptable means to i 
ensure not only desegregation, but also eauity of results; an opportunity 
to ensure community involvement in policy-making; and a means to intro- 
duce various changes into public schools. <. 

The first and possibly last time most of these objectives were in- 
corporated into one project was the Texarkana Drop-Out Prevention projects. 
O.EcO, essentially used performance contracting as a vehicle for evaluating 
the effectiveness of alternative learning systems used by contractors. It 
gave lip service to the turnkey notion, mostly to entice end»of-year coop- 
eration and data reporting from each of the districts; even though an in- 
il^i effort was madj^ to. .determine the im pact.'of -p erform ance -con tracting- ^of 
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and 1*11 explain later. Performance contracting vras' often perceived and 
sometimes advocated by O.E.O, officials as an end in itself, not as a means 
for reform, thereby adding further fuel and creating a blaze of controversy. 

Similar to the O.E.O. experiment, U.S.O.E. used the offer of incentives -I 
to increase student performance and to determine what parents and teachers 
would do differently; however, the research emphasis of the project did lot 
receive the attention it should have in the media. On the other hand the 
Texarkana project was a demonstration, not an experiment, of the concept as 
applied to the local situation and problems facing Texarkana schools, even 
though it was Federally-funded, 

What was the impact of these projects, particularly the O.E.O. project, 
on the performance contracting- turnkey movement? 

First, while the Texarkana project received the most "threatening" 
type of headlines (e.g., "firm guarantees to do what teachers can't), 
because of its local derivation, support by teachers and the involvement 
of so many in the planning process, knowledgeable and responsible teachers 
and other groups were able to see the real intent of the project . . . 
a catalyst for school reform. O.E.O. 's involvement in performance con- 
tracting changed many perceptions. The "threat" became credible, not 
only because of the size of the experiment, but also due to the fact that 
O.E.O. not O.E. took the initiative. The first NEA position stated by 
Mr, John Lumley before Congress expressed greater concern over this fact 
than over performance contracting. The President of the UFT, who had 
discussed the advantages^of contracting with teacher groups 3 months 
previously, sent telegrams to all Congressmen, and the President, calling 
for a Congressional injunction. O.E.O. 's Director, Donald Rumsfeld 
responded with a vitriolic attack in September, 1970, when most projects 
were being implemented, on organized teachers. The die has been cast. 

Second, in justifying the experiment, O.E.O. »s "advocacy" increased 
expectations boyond that which was possible. Contractor's naivete' in 
this regard also fostered unrealistic expectations. Performance contract- 
ing was never meant to be a "panacea". 

Third, by focusing primarily on student achievement, (as opposed to 
change) O.E.O. left itself open to criticism by critics opposed to exist- 
ing test insti^uments (e.g., nationally-normed standardized tests), a de- | 
f iciency existing before the. contracting movement began. 

And last,- the preliminary release without supporting data for ^-8 
months most assuredly caused retrenchment generally, although those who 
were intimately knowledgeable about the O.E.O. experiment and other attri- 
butes of performance contracting, did continue their plans, albeit with 
trepidations. 

Project Design 

The major criticism of the O.E.O. project from the education research 
and evaluation profession and GAG has been directed at its evaluation design 
which was determined by OEO staff. The evaluation contractor,, BMI, was 
responsible largely only for its impl^imentation. Several aspects of the 
design should be emphasized, ^ 

First, the use of a classical E vs C design is open to question. In I 
any controversial project, such as performance contracting, which was 1 

■.thEeaiten±n2:^to-^maiLv^tefl:&heaJs:;:^t.h«xJ!:Tvahni-;^ i2 
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prevalent, if not more so ihan any "Hawthrone" effect. Control teachers, 
especially m large Northeastern and Western "unionized" districts, appear 
-iu ^n^f increased student performance significantly as* they worked against 
the "steel driving machine" of performance contracting. In Seattle. C 
students achieved 1.5 grade gain. u 

Second, selection of firms was done rather quickly. OEO sent an RFP 
to a number of firms giving them 10-15 days to respond with a proposal. 
An addemdum was later sent to the firms requesting additional information 
regarding costs and levels of guaranteed performance. As a result, many of 
the firms implemented programs which in some cases differed radically from 
those. originally proposed. Hence, the ability of comparing technology- 
bases, vs mcentivebased vs other configurations was confounded and almost 
impossible for evaluation purposes. Ironically, materials from one firm, 
whose proposed guarantee was too low to be considered, were purchased and 
used extensively by several contractors by the middle of the project. 

Third, testing was extensive. Three standardized tests plus mastery 
of performance objectives were used for payment determination purposes. 
An additional standardized test was used for -evaluation purposes. .Because 
of possible Congressional criticisms that firms would teach faster learners 
at the expense of slower ones, OEO required each firm to negotiate a level 
of individual performance below which no payment would be made... an impos- • 
sibility for any firm to receive full payment because of the error of stand- 
ardized tests. Most of the test "overkill" resulted from the allegations 
in Texarkana, since disproven. 

Implementation 

Where performance contracting has been failed most seriously has 
been during project implementation. 

. First, planning time was inadequate not so much to do the necessary 
paper work but rather to minimize people-related problems. In the OEO / ' 
project, the districts' personnel met the firm's representative for the 
first time in late June and negotiated in less than one day a draft con- ■ 
tract which neither had previously seen. Busy ordering materials and de~ • 
veloping performance objectives, the firms had little time to cultivate I 
a good working relationship with principals and teaching staff before school S 
opened. Personality problems in some instances led to disaster, and bodily ^ 
harm. » ^ I 

I 

Second, the rigor of the evaluation design and the testing overkill 
constrained the ease of project implementation. For example, as BM J noted 
in its report, pre-testing conditions were inadequate in 2/3 to 3A of the 
sites. Principals who had to reschedule classes never forget nor forgave 
the project. In another instance, where the children in the X school were 
unexpectedly "disbursed" to several other schools due to a court order, 
OEO required the firm to establish learning centers in each school rather 
than selecting the next lowest ranking school for X. And even in those, 
instances where the firm and the district agreed to contract modifications, 
given the reality of the situation, such modifications were often precluded ' 



Third, prompted by NEA and AFT on-site visits by "truth squads" and 
national encouragement to resist, teacher resistance in terms of subtle 
Sabotage or strikes and even threats prevented delays in project start-up 
and in some cases laid^the foundation for failure. For example, .in the 
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of participating teachers disclosed that of the 215 respondents, 66 
favored the concept, 121 disliked it, and only 25 were undecided. OEO 
ordered a stop work, on this task shortly after preliminary results were 
compiled. To my knowledge OEO, which has these data, has neither con- 
ducted further analysis nor released many of these findings. 

Based upon a preliminary analysis of the OEO results and a review of 
our project documentation system I offer the following hypotheses: That 
the success or failure at each sit e can be attributed more to the degree tq 
which "interface" problern s existe^^. more than any other factor . As you will 
note in Chart I, in 8 sites experimental groups did better than control 
groups; and in 10, the reverse occurred. In the 8 "successful sites", 
the interface problems occurred in 7 of 32 possibilities while in the non- 
successful" sites, there arose 30 to kO possibilities. I asked Tom Glennon, 
now NIE Director, during the 1972 press conference, whether OEO would be 
analysing the above. No such activity has been conducted by OEO although 
some-OEO staff are developing a "white paper on the project." 

The USOE incentives project was plagued by similar problems. First, 
the projects were not officially approved until it was half complete 
precluding the hiring of project directors on-site and collection of 
baseline data. Second, contracts, with teachers were changed at least 7 
« times and were not signed until February or March. Parent contracts in 
some cases were not signed until after the completing of the project. 
Knowledge about the incentives project was not known by many parents and 
the projects' credibility among teachers varied. Third, the design of 
the project, especially the notion of paying individual teachers and 
Parents based upon student performance, cut against the grain of strong 
mores and ethical considerations. 

. My conclusion based upon these two projects is that the failure lies 
not with the technique of performance or even incentive contracting but 
rather with the ability of Federal agencies to design and implement con- 
troversial experiments. The USOE Incentive Project Evaluation Report 
written by ano;ther firm concluded that USOE is not organized to effectively 
implement such experiments. On the other hand, while OEO had the capacity 
and flexibility, some have argued that its naivete* and intent precluded 
any success, with performance contracting. 

Did OEO intentionally fail performance contracting? 

As you are probably aware, there are many theories and rumors why 
OEO treated performance contracting in its preliminary report, the way 
it did. 

On one hand, certain individuals, such as Jim Mecklenburger, now 
Director, Research National School Boards Association, and newspapers, 
such as the editorial staff of the New York Times, as well as others, have 
proposed somewhat of a conspiracy motive. 

Since the results? were not uniformly good, some argue OEO had much 
to gain by aggregating; grade levels such that the success would be cancelled 
by the nonsuccessful site. This would placate NEA resistance on the-Hill, 
since OEO's "continuing resolution" was being acted upon and indeed was 
passed four days after the results, were released. Leading statements were 
made by OEO official-S' such as Dr. Tom Glennan, when asked did teacher re- 
sistance -affect the project in any way, his reply was, "I don't think so." 
Representatives from certain firms, as well as GAO argue that the project 
•was -so .-poorly concel-ye-^d. -tha t- i-t ^precluded, .-pe r £ormanc eucontEacitine^toi-^su a aaaf^-. 
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While I'm sure that all of the facts have not surfaced, I offer 
the following observations. ' 



1. 



2. 



The experiment which was given top priority during initial stages, 
received the lowest possible priority a year later. The reason is not 
clear. However, it is true that Director Hujnsfield, and Assistant 
Director, John Q. Wilson, (both strong supporters) left OEO and the 
Project Director was taken ,off this project and put on another. Also, 
during mid-year, interest in the vouchers program began to receive highe^ 
priority, and some of the staff associated with the performance contract 
experiment moved over to the vouchers program. This lowering of priority 
made things difficult for both the school districts and us* as a manage- * 
menc support group. -r For example, we were using the OEO Computer Center 
at OEO's direction to develop a cost-analysis model. In February, OEO 
fired the contractor who was doing all of our, programming and data re- 
duction, but never replaced the firm. And OEO staff at the center gave 
the task low priority setting our cost analysis back four or five months,- 

It became clear as the project got underway, there existed naivete' on 
the part of OEO officials regarding school system operations. And to my 
knowledge, the original OEO Project Director never did make one site ' 
visit to one performance contract school building. Lack of under standint 
about the nature and extent of operational problems created largely by " 
the conflict with the evaluation design, could therefore not have 
been expected. 

As one might naturally expect, the controversial nature of the experimenii 
put OEO m a very threatened position, and one could normally expect, 
those whose vested interests which were being threatened, were trying 
to get at OEO. Hence, some of the considerations and decisions made 
toward the end of the project reflected concern about protecting OEO, the ' 
experimental design, and its activities, rather than having the intent ' 
of tiie project upper most in their minds. For example, mention of site ' 
visits to our offices or to the sites by GAO personnel, and visits by l 
Congresswoman Edith Green's staff, aroused fear more than once. 

Having attended the OEO PRESS Conference, when it released its prelimina' 
results and subsequently having talked to officials involved in preparing 
the preliminary report, I am convinced that the release was badly handlec 
Even though OEO was under a court order to release the results of the ' 
Seattle site by February 1st, the results could have been reported in 
a more professional manner. For example. Director Sanchez did not read 
his prepared text, but rather spoke off the cuff, setting the founda- 
tion for a very negative discussion later. Even though eight of the 18 
experimental groups did better than control groups, this was played down, 
and was included in the last part of the preliminary report. There 
were no statements of the difficulties of program implementation, pre- 
tests conditions, etc., as stated in the Battelle Report, which was 
available at that time but had not been released to the public. 

In summary, I am inclined to agree with Professor J. K. Millers' 
February article in Phi Delta Kappan entitled "Not Performance Contract- I 
ing. But The OEO Experiment Failed", not so much because of the poor 
evaluation design but rather the nximber of interface problems which 
arouse due to the conflict between the design and the ease of project 
implementation. And based upon my experiences with OEO, OE, and other 
Federally-sponsored experiments with promising yet controversial tech- 
niques, I shudder at the thought of further Federally-directed and— 
sponsored e xpjr^men,ta,fel&S .. mt.il. ijb. dexelopa .tha 'Iknow-how" .and-.csipacij:y 



